Training Deep Models for Semantic Segmentation

This topic provides information for generating and training deep models for semantic segmentation. In many cases, Deep Learning has surpassed other approaches, such as thresholding, K-means clustering, classic machine learning, and others, for semantic segmentation. Semantic segmentation is the task of classifying each pixel in an image from a predefined set of classes that are “semantically interpretable” and correspond to real-world categories. This is also known as dense prediction because it predicts the meaning of each pixel.

Semantic segmentation is different from object detection (see Object Detection with YOLOv3) as it does not predict any bounding boxes around the objects. As such, different instances of the same object are not distinguished. For example, there could be multiple fibers in an image and all of them would have the same class.

The following video provides an overview of image segmentation with Deep Learning. Other videos in this section describe case studies about training deep models for different segmentation challenges.

The video below provides an overview of training deep models for segmenting image data.

Image Segmentation with Deep Learning (43:05)

You also view this video and others in the Getting Started with Deep Learning section on our website (www.theobjects.com/dragonfly/deep-learning-getting-started.html).

The following topics are discussed in the video. Links to Help topics and other sources with further information are also provided.

The video below provides a case study for segmenting fabric fibers with Deep Learning.

Training a Deep Model for Segmenting Fabric Fibers (23:16)

You also view this video and others in the Deep Learning Case Studies section on our website (www.theobjects.com/dragonfly/deep-learning-case-studies.html).

The video below provides a case study for segmenting the multiple material phases.

KitKat Segmentation Case Study (11:44):

You also view this video and others in the Deep Learning Case Studies section on our website (www.theobjects.com/dragonfly/deep-learning-case-studies.html).

The following items are required for training a deep model for semantic segmentation:

  • Training dataset(s) for the input. See Extracting New Images from Marked Slices for information about extracting a training dataset as a subset of the original data.
  • A target for the output(s), which must be a multi-ROI(s). For each class in the multi-ROI, voxels must be labeled in a semantically significant manner (see Labeling Multi-ROIs for Deep Learning).
  • A model that supports semantic segmentation.

    Note A selection of untrained models suitable for binary and multi-class segmentation are supplied with the Deep Learning Tool (see Deep Learning Architectures). You can also download models from the Infinite Toolbox (see Infinite Toolbox), or import models from Keras.

The following items are optional for training a deep model for semantic segmentation:

Multi-ROIs that are used as the target output for semantic segmentation must meet the following conditions:

  • The multi-ROIs must have the same geometry as the input training data.
  • All voxels contained within the input patches must be labeled to be considered during training. Patches that are not fully segmented will be ignored.

    Note Applying a mask may limit the number of input patches that are processed (see Applying Masks).

  • For each class in the multi-ROI, voxels must be labeled in a semantically significant manner.
  • Whenever possible, an equivalent number of voxels should be labeled for each class.
  • Training is always done on the image plane and all classes must be labeled on that plane.

As shown in the illustration below, three distinct material phases in a training dataset were labeled as separate classes. You should note that labeling can be done directly on a multi-ROI as of Dragonfly version 2020.1 (see Classes and Scalar Information). You can also choose to work on multiple regions of interest, from which you can create a multi-ROI (see Creating Multi-ROIs from Regions of Interest).

Multi-class labeling

Multi-class labeling

You can use any of the segmentation tools available on the ROI Painter and ROI Tools panels to label the voxels of a multi-ROI (see Classes and Scalar Information). You should note that training is always done on the image plane and all classes must be labeled on that plane.

To help monitor and evaluate the progress of training Deep Learning models, you can designate a 2D rectangular region for visual feedback. With the Visual Feedback option selected, the model’s inference will be displayed in the Training dialog in real time as each epoch is completed, as shown on the right of the screen capture below. In addition, you can create a checkpoint cache so that you can save a copy of the model at a selected checkpoint (see Enabling Checkpoint Caches and Loading and Saving Model Checkpoints). Saved checkpoints are marked in bold on the plotted graph, as shown below.

Training dialog

Training dialog

  1. Scroll to a representative slice within your input dataset.

  2. With the Rectangle tool that is available on the Annotate panel, add a 2D rectangular region that includes the area or features you want to monitor during training.

  3. With the Rectangle tool that is available on the Annotate panel, add a 2D rectangular region that includes the features you want to monitor during training (see Using the Region Tools).

    Note Any region that you define for visual feedback should not contain labeled voxels.

  4. Choose the required region in the Visual feedback drop-down menu on the Inputs tab when you set the training inputs.

    Note The visual feedback image for each epoch is saved during model training. You can review the result of each epoch by scrolling through the plotted graph. If the checkpoint cache is enabled, you can also save the model at a selected checkpoint when you review the training results (see Loading and Saving Model Checkpoints).

Dragonfly's Deep Learning Tool provides a number of deep models — including U-Net, DeepLabV3+, FC-DenseNet, and others — that are suitable for binary and multi-class semantic segmentation. Semantic segmentation is the process of associating the voxels of an image with a class label.

  1. Choose Artificial Intelligence > Deep Learning Tool on the menu bar.

    The Deep Learning Tool dialog appears.

  2. On the Model Overview panel, click the New button on the top-right.

    The Model Generator dialog appears (see Model Generator for additional information about the dialog).

  3. Make sure that only Semantic Segmentation is checked for the Filter on model type options.

    This will filter the available architectures to those recommended for segmentation.

  4. Choose the required architecture in the Architecture drop-down menu.

    Note A description of each architecture is available in the Architecture Description box, along with a link for more detailed information (see also ).

    Recommendation U-Net models are often the easiest to train and produce good results in many cases. For a more powerful option, you might consider Sensor3D, which uses a 3D context.

  5. Choose Semantic segmentation in the Model Type drop-down menu.
  6. Enter the required class count for the neural network in the Class Count box.

    For example, if your training set multi-ROI has four classes, then you must enter a Class Count of 4. If you are training a model for binary segmentation, your multi-ROI would have two classes and you must enter a Class Count of 2.

  7. Enter the required number of inputs in the Input count box. For example, when you are working with data from simultaneous image acquisition systems you might want to select each modality as an input.

    Note If required, you can edit the input names.

  8. Choose an Input Dimension as follows:
    • Choose '2D' if you want limit training to 2D, i.e. slice-by-slice.
    • Choose '3D' and then a number equal to or greater than '3' to train in 3D, in which case multiple slices of the input dataset will be considered for the output target (see Configuring Multi-Slice Inputs).

  9. Enter a name and description for the new model, as required.
  10. Edit the default parameters of the selected architecture, optional.

    Note An initial set of training parameters, which should work well in most cases, are provided with the semantic segmentation models supplied with the Deep Learning Tool. You should also note that training parameters are saved with each model and can be reused for future training.

  11. Click Generate.

    After processing is complete, a confirmation message appears at the bottom of the dialog.

  12. Close the Model Generator dialog.
  13. Select the new model in the Model list and then click the Load button, if required.

    Information about the loaded model appears in the dialog (see Details), while a graph view of the data flow is available on the Model Editing panel (see Model Editing Panel).

  14. Continue to the topic Training Deep Models for Semantic Segmentation to learn how to train your new model for multi-class segmentation.

You can start training a supported model for semantic segmentation after you have prepared your training input(s) and output(s), as well as any required masks (see Prerequisites).

  1. Open the Deep Learning Tool, if it is not already onscreen.

    To open the Deep Learning Tool, choose Artificial Intelligence > Deep Learning Tool on the menu bar.

  2. Do one of the following, as required:
  3. Select the required model from the Model list and then click the Load button, if required.

    General information about the model appears in the Details box, while the Classes box lists the number of classes selected for the model (see Model Overview Panel).

    Note You can edit the color and name assigned to each class. You should also note that the data flow through the model is shown on the Model Editing panel show.

  4. Edit the selected model, if required (see Model Editing Panel).

    Note In most cases, you should be able to train a semantic segmentation model supplied with the Deep Learning Tool as is, without making changes to its architecture.

  5. Click the Go to Training button at the bottom of the dialog.

    The Model Training panel appears (see Model Training Panel).

  6. Do the following on the Inputs tab for each set of training data that you want to train the model with:
    • Choose your input dataset in the Input drop-down menu.

      Note If you chose to train your model in 3D, then additional options will appear for the input, as shown below. See Configuring Multi-Slice Inputs for information about selecting reference slices and spacing values.

      Note If your model requires multiple inputs, select the additional input(s), as required.

    • Choose the output multi-ROI in the Output drop-down menu.

      Note Only multi-ROIs with the number of classes corresponding to model's class count and the geometry of the input dataset(s) will be available in the menu.

    • Choose a mask in the Mask drop-down menu, optional. You should note that in cases in which only a limited number of slices are labeled on the target multi-ROI, a mask will be generated automatically to include only those slices (see Applying Masks).

    The completed Training Data should look something like this:

    Note If you are training with multiple training sets, click the Add New button and then choose the required input(s), output, and mask for the additional item(s).

  7. Do the following, as required.
  8. Click the Training Parameters tab and then choose the required settings.

    See Basic Settings for information about choosing an input (patch) size, epochs number, loss function, optimization algorithm, and so on.

    Note You should monitor the estimated memory ratio when you choose the training parameter settings. The ratio should not exceed 1.00 (see Estimated Memory Ratio).

  9. If required, check the Show Advanced Settings option and then choose the advanced training parameters (see Advanced Settings).

    You should note that this step is optional and that these settings can be adjusted after you have evaluated the initial training results.

  10. Click the Train button.

    You can monitor the progress of training on the training Model dialog, as shown below.

    Model Training dialog

    Recommendation During training, the default =quantities 'loss' and 'val_loss' should decrease. You should continue to train until 'val_loss' stops decreasing. If required, you can increase the number of significant digits to make sure that small changes to monitored quantities can be noted (see Selecting the Views Preferences).

    When training is complete or is stopped, the Training Results dialog appears.

  11. Evaluate the training results, recommended (see Evaluating Training Results).

    You can also generate previews of the original data or test set to further evaluate the model (see Previewing Training Results).

  12. If the results are not satisfactory, you should consider doing one or more of the following and then retraining the model:

    Note If your results continue to be unsatisfactory, you might consider changing the selected model architecture.

  13. When the model is trained satisfactorily, click the Save button to save your deep model.
  14. Segment the original dataset or similar datasets (see Applying Deep Models).

    You can also process datasets in the Segment with AI panel (see Segment with AI).

After training is complete or stopped, you can evaluate the results in the Training dialog, as well as generate previews of a test set or the original data to evaluate the predictive model (see Previewing Training Results).

You can view the results on the Graph tab, shown below, on which epochs and the selected metrics are plotted on a graph. In addition, the inference within a selected 2D region is shown for each epoch if a region for visual feedback was selected as a training input.

Training results (Graph view)

Training dialog

You can do the following in the Training dialog on the Graph tab:

  • Scroll through the plotted graph and review the inference for each epoch, if visual feedback was selected as a training input.
  • Modify the opacity of the inference and input, as well as zoom and pan the saved checkpoint image by dragging with the left and right mouse.
  • Change the range of epochs that are plotted with the Epochs sliders.
  • Select the metrics that are plotted in the Metrics drop-down menu.
  • Export the results for each epoch as an image in the PNG file format.
  • Load a selected checkpoint, if the checkpoint cache is enabled (see Enabling Checkpoint Caches). You can then save a copy of the model at the loaded checkpoint (see Loading and Saving Model Checkpoints).

You can also view the results on the List tab, on which the learning rate and other metrics are shown in a table view for each epoch.

Training results (List view)

The results for the total training can be exported in the comma-separated values (*.CSV extension) file format.

Training results can also be reviewed for any model listed on the Model Overview tab. Right-click a model and then choose Display Training History and the required training log in the pop-up menu.
You can also compare different models in the Model Evaluation dialog (see Evaluating and Comparing Deep Models).